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Foreward n 1 

The literature on evaluation, both empirical and 



\ 

non-empirical, is extensive. 1 ©f course, as Cook (1978, 
p. 14) and Shslpiro " ( 1 979„ P 1 ) j?ote , . the quality of the 
literature 1 varies greatly. Nevertheless, the literature is 
at least extensive enough to- b^in to-support. stfrith.esis of 
knowledge we have gained t&g a date, regarding evaluation 
^Thompson & King\ 1981b). Furthermore , as -Thompson and'King 
(1981a, p. 20) observe, w%„Uow Jiave at least the beginnings 
of a base of , theoretical^ ^propositions- about evaluation; 
these can now be used to feuide , at le^t to some extent, the 



efforts of evaluator^s ancfVvaluat^on • researchers 



Despite the -fact J^t the evaluation literature is 
extensive, it does cont,a$,n one noteworthy gap: the views of 
LEA • evaluators tend Mcl be uader repr esented in - the 
literature. .For eica^Le # , evaluation' journals , such asCEDR 
Qua rterly and ,EdUcalt#nal Evaluation and Policy Analysis , 
infrequently cont$i#/* articles written 'by LEA evaluators. 
Fortunately,' Staled . (1 980) has provided a' glareing 
exception to thifprule. On behalf of NXE, he has edited a 

\ • :Sf ' - . • • : . 

Set of. papers wr%%n by five teachers, five principals, and 
five v LEA ev^Jators," regarding their perceptions -of 
evaluation and listing in schools. #he . compilation was 
ori-ginally disseminated by the U.S. Department of Education 

m - - : ' • " ' ' ■ 



Page ii 

as a paperback, and is now also avail-able from ERIC. ■ 

This' symposium was organized to fi^Ll, at least partly, 

the gap which 'stil* exists in the evaluation literBt^ure.^ 

The symposium was grounded in a' premise that LEA evaluators* 

have important insights to. add to the knowledge gained from 

more formal naturalistic and simulation ^studies of the 

^evaluation \ process. Consequently, • the LEA symposium 

participants were selected to represent diverse backgrounds 

v-is a vis geographic location, -LEA enrollments, and the 

organizational^ framework in which- each evaluation department 

operates. .*The nbn-LE^.par ticipants were "also ^selected ^tq 

represent diverse backgrounds vis a. vis experience andi 

' - . * V 

evaluation perspective. 

The symposium was presented in- an innovative -format. 
V * . 

The non-LEA symposium" participants!' were asked to pose 

' written questions regarding "Evaluation from /an, LEA 

* - • *. 

Perspective." The questions were mailed,' to the; LEA 

participants well in advance' of the 1^82 AERA^ annual 

meeting. This' document represents the proceedings of» the' 

symposium insofar* as it presents the' LEA- , e.v?luat}or 1 s 

responses to the posed questions. Both the questions aad 

the responses were- . developed independently by S^ach 

participant, and* served as the basis, for more 'extended 

* * / - . * . \ 

discussion during the annual, meetings. . . t 9 ' 
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J (Questior^r: CAROL H. WEISS) v 

1 . How much, of what' LEA evaluators do is "evaluation" in the classic sense , 
the observation of specific outcomes. of discrete educational programs? 
»• ' • else do LEA evaluators do? , » 

/.(.Respondent: STEVEN -FRANKEL) 

In air case, classic evaluations account "for about 25% of our budget. *An. 
additional* 25% is allocated, towards management studies of the noninstrUbtional 
aspects of the school system; about 15% goes towards standardized testing t 
prograM;' another 15% goes to. conducting fiscal -audits; " and the remaining^ 20% 
covers coordination of policy grid regulations development and the development*' of 
statistical J reports . > / - ' ' 

' .(Respondent: FREDA HOLLEY) ~ : " 

Most R and £ offices', do not really work on the jdiscrete project basis implied by 
this question. In. fact, . I have a hard time dealing with- this question from my \ 
framework. To me, evaluation* is looking at dita and making or helping educators ;i 
make decisions to. improve education. ' The Lybn and Doscher (1979) report,^ for - 
example r said that most of £he£& offices spent anAinordinate amount of ti^me* on 
"testing" with the implication that theywefe ndt aoing much evaluation. However, 
most ^evaluations done in school districts should and do involve - student * learning 
as an important program outcome measure.* Therefore, a district-wide testirfg 
program is likely to feed into the majority of all evaluations .conducted. Is - 
testing just testing' then or is it also an evaluation .activity? Recall bow much V 
'o^many major research projects such as those on teaching effestivenessHiave been- 
s devoted to collecting -su$h test data as a" part - of their project. No one 
•questioned whether this^was a research activity. In school districts it is both a 
way of. evaluating student? and programs and district achievement. . y 

- 1 With the testing phase of an evaluation', taken care of f evaluators then 'have most 
-of their resources .Or staff time available for other" activities of which there are 
multitudes, f .Of' course, " there . is classroom observation, interviews, 
% quest i6nna ires, .analysed, and "reporting; evaluators are involved in almost every^ 

activity that has or .can be" described as evaluation. t tyey also spend a good deal \ 
* \ of time in -our office in \ management- ' activities' since N we jjave differentiated 
st.affirtg. In our office, evaluators are administrators; they supervise staff, do 
, * personnel evaluations, order, supplies, and -so on.' Ihey spend a great. deal # df time 
j%st 'communicating other than .formal reporting purposes: writing memos, providing 
*^1ifiservac0, talking informally - to ^administrators, consulting. They spend ai 

* inordinate amount of" time on the telephone just answering qujesticfns. 

* • 
Since many evaluation activities airSo* involve record keeping' for various program 
c purposes, t this must also be 4 accounted for. t Thi$, like testing however, often 
'* ^ feeds into, the evaluation making it possible to do, more' complex analyses taking ' § 

into account the various factors^on which records, are maintained.- 1 

# Finally, most of us &Lso spehd.time just asking' ourselves questions* such as: ' W %t 
# : information is really important .to our district? Wha£"kinds of questions that' 
■ # ' matted should we be asking that we* are not? How doesHhis data* that^we have makfe 

sense .in terros of tti$ way tlie. system works , -of, the way learning occurs, of 'the .way 
a program. develops, of other national * research? 
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(Respondent: CARL NOVAK) - - 

Jn response to this question I have attached a midyear review of evaluation team 
'activities proposed for our Board of Education (see Appendix A). The review 
emphasizes two trends: a diversification oP the types of services that the 
evaluatipn -team provides and a trend atJay from evaluation of federally funded 
programs toward support of locally funded 'programs* Ihe two trends are related. 
The evaluation ■ t^ayn was initially organized to evaluate (in the classic sense) 
federally funded programs. As our focus changed so did the services we provide. 
We stil\ "evaluate" programs, but only if evaluation /questions are being asked. 
Other kinds of activities the evaluation team ^is involved in include the 
development, interpretation . and use % of educational tests,, community and staff 
surveys, support of curriculum studies,- informatiori processing and use, needs 
assessment, and any 'other Special" studies (which in a different setting could be 
called research) . A list of 45 projects on which the evaluation team was fcworking 
was compiled for the same Board meeting. Ihe list included 20 projects that are 
best described as evaluation projects. Ihe 20 .projects accounted for 50% of 
professional evaluation time flrpm- September 1, 1981 thru February 1, 1982. 

(Respondent: ELLEN PECHMAN) • , 

.1 think it has been a* well known but little acknowledged fact for some years that 
evaluators don't do much work at the program* level wh;Lch .follows the rational 
models established in texts. a When we have^ the luxury of flocking with a 
sufficiently evaluation-oriented and self-confident client, we are, indeed/ able 
to apprQach the process of objectively examining t programs in terms, of - outcome 
effectiveness; but this occurs too infrequently t Ihe discus^nts on this panel 
represent a range of school districts in. this regard. Some of us are from 
districts that have had, a Sophisticated research orientation, for quite some time, 
and evaluation is a mor'e accepted process. Others, like me, work* in districts 
whose history and politics make it difficult to undertake the self-analysis which 
can.be -conducted in educational climates that are more homogeneous or, at least, 
politically less volatile , I believe that systems are like people— the greater # 
.the* success, self-assurance and confidence, the more possible it is to s be 
self-critical 4 f 

But t i£ we dfcn't evaluate in* the traditional sense, wh£t ^ we do? Sixteen 
percentTtoi^my departmental budget* goes towards program evaluation and, of this, we 
ar%only evaluating 'one district initiated program. The evaluation staff * works 
almdst exclusively with externally funded programs and provides supportive 
technical assistance to school principals, project directors and grants *dev elopers 
, oh*. the ev&luatioh comment* in their proposal*. Nineteen percent, of our budget 
is tiirected toward the systemwide testing program which is evaluation, ljut in" the 
narrowest s^nse-. The bulk of the remaining budget supports* the*massive student 
dataV^che^ulin^ and mark reporting system in the district, and the 'census and 
state*" and, federal reporting activities which also emiriate v fVom my office.-/ 

* Irv the- past two years I have been able to put ~ some additional staff rresources 
Cthree part-time professionals, -equal to about 5% m of the budget) ^into hiring 
p^^sonnel who have* highly technical research and data analysis skills.. This gr # oupr 
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is beginning -to serve as a policy information support group for top management. 
This staff works with aggregate program data which are easily obtainable through 
the routine channels of system accountability or from quick surveys that respond 
to requests to conduct one-shot studies for top management. In a few rare cases, 
we- are also able to help -principals by. pulling the descriptive data about their 
school, such as its 'achievement profile, or we help produce descriptive data their 
teachers can use * for program analyses-. Among pur activities of this type are 
several systemwide needs assessments, one of computer needs and the- other of test 
needs ; a datfa based model for* selecting high need schools in which to establish 
experimental first grades; and, finally, an analysis of the test data pattern of 
a school which appeared to have inflated test scores. These pieces of research 
are very supportive to the system, but they are not very flashy and can hardly be 
called evaluation. Although evaluation in the tr.aditional sense is not often 
done, we think we have been increasingly , effective in providing important 
supportive data 'for managers to use in their planning and decision-making. 

(Respondent: FLORALINE STEVENS) 

My perspective can only come from my experience as an'evaluation administrator in 
a large urban school district. I would say that LEA evaluators devote about 
one-third of their time to "evaluation." A good portion of their time is spent 
with, /ederal "and state compliance "surveys' and the administration of testing, 
programs— norm-referenced y criterion-referenced and "competency. LEA .evaluators 
attend many meetings serving as resource persons. They write many directional 
bulletins to schools and offices about how to fill out survey forms, conduct 
testing and so forth! 
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(Questioner; CAROL H. WEISS) ! 

2. -What kinds, of impacts have evaluation* reports had on LEA policy and practice? 
What . conditions have been conducive" to 1 great.er impact? Fqr , example, has 
continuity of evaluator service bpen a factor, or personal relationships' 
between' evaluators .and ' decision makers, or ; oral informal conversations, or 
propinquity of office location, or salience of the ' issues addressed in 
evaluation studies? Is the^ scientific quality of evaluation an important 
factor in generating attention? 

(Respondent: STEVEN, FRANKEL) < H /* 

Our reports have had considerable impact in about one-third of the cases. .This 
impact has b^en evidenced by programs being expanded, modified or cut back largely 
on the basis of the results. ,In another third of ^the cases, the results and 
recommendations have bfeen largely ignored. In the^remaining third, the ' evaluation 
did not come up with findings which justified '.modifications in the 'existing 
.programg . j * 

T^e most important factors in determining our impact have been: 1) Our staff is 
regarded as being competent and evenhanded; 2) We report directly to the 
superintendent and are therefore butside the span of contrdl of any of the program 
managers; and 3) All of our reports are automatically transmitted to the Board of 
Education and made public. . i 

Scientific quality must be a given in that your constituents must 1 be confident 

fcthat your reports can be subjected to outside scrutiny, In most cases, we will 

not be asked to do «a study unless the topic is very controversial. This means no 

matter what we find and* recommegd, there will be a considerable body, of people who 

look to discredit the results 

* 

• 1 ♦ 

However, Elaborate statistical techniques or designs are often: counter-pFoductive . 
Their Impact ' on cost and timeliness qre seldom justified by what they add to a 
study. For the most part, we rely on straightforward procedures yhich. can be 
conducted w^th a minimum of risk. This is because, in addition to ^accuracy, the. 
next most desired attribute is timeliness. The best % study in the world is useless 
if it surfaces after the Board or County Council has voted on a program.' s future., 

* . « • 

(Respondent:* FREDA HOLLEY) . J 

• Perhaps the best description of the sometimes difficult to observe link between . 
evaluation and practice \ have heard .came not from'*an evaluator,' byt from A 
Mexican American program administrator Sk\er\ asked a similar question in a 
presentation about Title I evaluation/ He said that now when he walks intq a 
classroom 0*>a school with a low ^socio-economic population in. just about any 
school in our state and asks how the children are doing, he isn't told anymore 
*"just fine." Rather ife receives specifics in <terms of test scores and growth' from 
the last year to th£ present. When he asks if the Title I program- is helping*, he 
gets "specific answers also about average growth, hours bf service, and types of 
programs. * . ' 

. ' ' N 

What the administrator is saying and what I hav^ observed is that you see" the 
change over time; in retrospect the change' may be dramatic, but from close-up -you 
may not even be aware of it. Evaluation does make a difference but capturing the 
difference is difficult. There are times, of course, when you do see immediate 
improvement. I tried to # give some examples, in my^ article in the* Stalford. 



f 
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collection Gsee bibliography in the Forward of these proceedings) 



I have also said at various times that the impact an evaluation is likely # to have, 
may "be due to an array of .factors, the „ most significant of which ark the 

importance and relevance of the evaluation findings, first, and to the 

perseverance and salesmanship of the evaluator, second .i 

Although "scientific quality" per se would not be the way I wobld "phrase Lt,^ the 
certainty of the finding does play, an important role in generating attention and 
, inspiring confidence for utilization ancLthat is associated with the tightness and 
quality of the -design.. I have been asked if evaluator s really need sophisticated 
statistical skills. The assumption behind this question is that most ' evaluation 
•work is not very sophisticated.- But I belfeve that is .the chicken-egg dilemma. > I 
find that where staff have more sophisticated skills, these skills get execcised.' 
I think we in school, evaluation continuously face vef y complex questions— f>r more 
complex than are encountered in laboratory situations— whose i. answers 'we cannot 
supply or supply inadequately either because our skill or the state-of-the-art. 
| itself is inadequate to the challenge. ^ 

(Respondent: J3ARL NOVAK) \ . . - 

Unfortunately "the . scientific Quality of a particular ^evaluation is not an 
important factor in determining -the perceived value of the evaluation. More 
/important factors .are the salience of the' programs and issues addressed, the 
extent to vdiich the. results focus on what are perceived to. be the-key aspects of 
the program, the extent to which procedures -are perceived as 'fair (not biasedly 
representing any one point of view), the personal credibility of the evaluator, and 
• ; the credibility of- the evaluation unit. 

• . ■ ■ Q . ■ . . ■ . ' 

The professional relationship between the evaluator and decision maker is % also 
important (much more so than a "personal relationship). In my estimation the. most 
effective format for presenting results and maximizing the potential use of 
Cesults is a short informal presentation by the evaluator to a group of key 
decision makers (two or more) followed by unstructured discussion 6f the program 
> and the issue surrounding the program. This format allows decision makers to 
"focus on those aspects of the program that they feel are most important (both pro 
and con). The role of the evaluator is to relate the results of the evaluation t@ 
those issues. ^ 

(Respondent: ELLEN PECHMAN) 

. In our case, I must candidly acknowledge ftat evaluation reports,^ as such, v have 

had -little or /any direct impact' on policy or practice in. the district. 

Ironically, I think the most interesting and most direct impact evaluation has had 

is in molding behavior and encouraging accountability through Vn m threat of 

'evaluation. This is instrumental evaluation in >a- very unusual sense !^|f 

V 

Most of the ^evaluations we have conducted over the years have been- mandated 
reports which describe, in routine format and carefully wooded neutrality, the ' 
superficial elements of the* programs ,*i.e . , the number- of participants, whether* 
meetings' were held as planned, that parents or. teachers came to meetings, and 
reports of 'test scores. However, we have several notable ^xamples of programs* 
getting . underway several months earlier than they might have, had the, evaluator 
* not proded the system. Furthermore, several programs have been quick to report 
their own data to Board members and superintendents to prove they didn't need to 

O ■ 1 
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be equated! "\ 

This year^I think, we will have the first" o$por tun itjr in *a long time to provide 
direct evaluative assistance for planning to an important districtwide program. 
Not surprisingly, this is a .sensitive and controversial program in the district 
which most pebple think the superintendent would lil^e to see disappear . *0ur data 
on this programVLargely consists of 'reports from teachers, administrators and* 
students. fcxcept. for the results of a normed test of basic reading and math 
skills, we still- have been unable to obtain sufficient access to the pcpgram to 
collect observational data on behavior or to conduct pre- and* post-tests <Jf 
specific skill achievement, among students?* * Nonetheless, the kind of descriptive 
reporting we are < doing has hot been done before in our district and I think it 
will be helpful to program managers \rtio are planning for next y§a^. 

To answer Carol Weiss v qu^tion more d^refctly, I 1 think all of the factors she 
jnentioned in her questions are " important /to .the success of evaluations and 
evaluation relationships. Eacti one will be more or less important depending v upon 
the politreai^and program v oon£ext, and the people involved. In my experience, the 
issue of scientific quality lis probably the least important factor in the • 
-evaluation process or product* Technical quality is' an issue for our peers in the 
evaluation communi'ty but, in XStie school setting, it is rare that evaluation users 
are professionally interested or experienced enough to be very analytical about 
the technical elements of an /evaluation . Regardless of- how' technically corVe^t or 
effective an evaluation is, it will only be used if the results coincide with the 
intuitions of- the significant decision-makers or constituencies. Of course, the 
^report for each audience (user) is absolutely essential, but I also think that it 
is 'relatively easy to design^ reports to meet users 1 needs. I£ gust requires that 
the evaluate^ be sensitive and * open to "lay" suggestions about formating, and 
writing up findings. * * • ' * * 

One factor affecting imfact of 'evaluations in LEA's which was not mentioned in the 
question is administrative'* clout . • In some ways this is obvious, but I haven't 
seen- it discussed directly in the literature. The rational {nodel ^of evaluation 
suggests that-clout will Gome with evaluation quality, but we don't think this is 
the case in so highly a. political and ^bureaucratic 'environment as schools. Clout 
is both a personal and an administrative phenomenon arid many outstandingly skilled 
evaluators will lack the clout , even to initiate* and/or successfully follow through 
on the* evaluation process .itself. Without access to a program, «it is difficult to 

• obtain the basic data needed for its evaluation. Thus, I would say ' that strong 

* administrative support j.s a critical element of evaluation report impact in a LEA. 
This ppwer of good old fashioned clout comes from a just right match ^o.f skills, 

'circumstances, context; experience, -technical expertise, personal relationsHips 
and, sometimes, even luck! t 

(.Respondent^: FLORfiLlNE STEVENS) * * * ' , 

The impact of evaluation reports depends on what has been * evaluated. The 

♦ perfunctory Compensatory Education/Title, I Reports have had no impact on LEA 
policy and practice; hpwever , £his # wiir~crfange*because the content of the report 

' was' changed. We are now conducting an ethnographic study of successful reading 
results in- Title I schools. LEA policy person^ have expressed much interest in 
the outcomes. It is anticipated that the information will be widely disseminated 

'and used for staff development. Any evaluation that provides insights into 
successful teaching practices 'has great, value because our new superintendent has 



five goals for improving the .schqol distribt, ancf JpLgbfei* . achievement will be sn , j 
index to indicate better achievement among' the st'uderitsT Ihere is atMexcitement 
and a push in the LEA to promote and enhance .the conditions for -principals .Ao be % 
» • better instructional leaders, teachers 'to provide quality instruction / and -for £ 
students to learn nibre. .Because >of these elements, .any 'information that may prove 
helpful is sought and' is welcome/ ' As the evaluation reports in the district 
provide the kinds of information that ape needed and' are Useful, |fpey t wfll ,hav.e • 

• * impact. The LEA administrators have befcn in' the district long enough- to know and*.,' 
4 aid the top level administrators. , They noy seek .aid. from <the research ;and . 

* evaluation;" office. -The office is located in the administration* comple* ^aftd 
' therefore 'personnel 'and -materials --are • irrciiediateLy accessible* to LEA. 
• f administrators. Ihe . quality .of the information prjovided in court-orderfed 
evaluations or federal 'compliance, r'eporjt^ .have had impact. The politibal 

• fc ramifications of the- reports, 'not the .scientific quality f^Jiave' generated attention 
" ' although the scientific quality- (evaluation methodologyk has been excellent fq^ 

manfr of the reports. ' # V 



-a? 



. ' ~ . x , Page 8 

, «* » ' . 

(Questioner: G&ROL H. WEISS) . .1' 

3. What does the looming cutback in resources portend for the- future of LEA 
evaluation? If \budgets__Jaave to be cut, will evaluation- be among the first 
functions to be tenhTnated? 'Or will superintendents and others turn to 
evaluation to help in budget-cutting decisions? « 

♦ 

(Respondent: STEVEN FRANKED 

If evaluation groups configure themselves like us, with about ^half the resources 
going towards management studies and audits, the future is rather rosy." The riiost 
conservative boards and superintendents 'will support an operation if. it promises 
to return several times its cost in savings. Further, if the staff and resources 
ne'eded for the management studies and audits are within the span of control of the 
fresearch director, those same talents and techniques can be .applied to the 
instructional side of the house as well.. As evidence of this, our operation has 
been growing ' at about 10-15% a - year , despite declining enrollment 'and a 
conservative foard dedicated to putting every possible dollar into teachers and 
books*. * j i 

As intimated by tfie question , this .only occurs if the research operation is 
willing aid able to assist the Board and superintendent, in their budget cutting 
decisions by providing them with uiequivicable data showing which operations are 
performing as expected, and uhich are not. 4 # » 

From another perspective, the future of LEA evaluation units is even better. Ihe 
way things are going, we may soon be the only gaihe in town. 

The cuts in Federal support don't hurt us as they do the labs, centers, and 
consulting organizations because most of us never saw those dollars anyway; 
except to do studies the world can easily do without such as the Tftle I and Head 
Start evaluations. Similarly, we are now coming out of the period of enrollment 
declines, and therefore Hill not be suffering the same cutbacks as universities 
will begin experiencing as the drop ; in <the birth curve starts affecting them. 



Therefore, I think that there is a gocft ctonce that ,the school system based 
researon operations will become stronger and increasingly influential in the next 
dedad^ • 

(Respondent: FREDA HOLLEY) . . 

LEA evaluations will undoubtedly suffer along with the rest of education in _ the 
•current cutbacks. ".Evaluation, is rarely of any real help 'in budget-cutting 
decisions; we are far from the coverage, and exactness that would 'be helpful in 
fact. Our district faces a cut from fibout $3 million, in funds for desegregation, 
innovative programs, library services and so forth. Probably the highest priority 
.service for continuation at the moment is that of bus monitors. But this- is 
really a value judgment decision since v*io ean real 1/ do an evaluative comparison 
between bus monitors and ' .library books. Even if formal evaluation offices 
disappeared , however-, I personally believe that education is never § likely to be 
quite, the same as it .was before our hey day and that evaluation skills will 
continue- to be important in education. . 

m ■ 

(Respondent: CARL NOVAK) 
'This is a decision that will have to be made on a district by district basis. The 
outcome will (or at least should) depend upon the extent to which evaluation 
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results (or the services provided by the evaLuators) have become an integral part 
of the decision making process. If evaluation is perceived by. the Superintendent 
as window dressing, something \to be done primarily ,.fo'r public relations reasons, 
-then evaluation probably ' will be cut. However, if the services provided. by 
evaluators (however*they are classified) are considered to be useful and, cost 
'effective, 1 then evaluation will probably continue to be funded. ^ — 

Our evaluation uii£ has been fortunate in that the Superintendent and- the Director 
of the Educational Service Unit #18 Have both been extremely supportive. Decision 
making within the district is decentralized and the district is structured to 
allow the free flow of .information. However, the Superintendent-- recently 
announced his 'retirement.' The* change in the super in tendency coupled with 
increased fiscal pressures caused by a -State lid and further reduction in federal 
programs are going to make* the next 12 to 1$ months interesting. At this, time I 
am still confident in the future of the evaluation unit in our district, however 
we are currently making a concerted effort to make sure that the evaluation team 
activities are consistent with district priorities . 

(Respondent: ELLEN PEOHMAN) 

In our district, we are not sufficiently geared up as an evaluation unit to 
provide concrete evaluation data for budg^ cutting purposes, as such. In the 
next yeaf or two, however, if we have the opportunity to build our .management- 
information system, it is my hope that a good deal of planning data will be 
available to enable us to analyze where the system's needs are greatest and where 
funds and .support can be best spent. This is more data based planning than 
evaluation, but the tools of the- evaluation researcher are obviously critical to 
the effective use of a management information system. 

Regarding the use of evaluation reports as the basis of budget cutting decisions, 
-my^concern is that this is a very negative use of evaluation—and one which should 
be avoided . If evaluation processes and reports become instruments for budget and 
program surgery, program people understandably develop masterful means of avoiding 
the operation entirely! Already too many people think of evaluation ,in a negative 
sense and, if it is used- maioly as the tool to* eliminate programs, we will 
continue to have a hard time eliciting people's commitment and participation in 
the evaluation process. / % 

A ' 

.As I write this response, vk are in budget time in the district and my proposed 
budget includes requests ' for staffing and testing increases. It will be t 
interesting to see 'the 'reaction of the budget committee. Our recently appointed- 
superintendent 'has expressed a verbal commitment to planning and evaluation, but 
it is unclear if he -appreciates what good,' evaluation requires * in manpower, 
dollars,, ana, most important, 'in administrative/ commitment and oversight. My 
experience is that many people who support evaluation do so fVom the arm chair and 
the translation of support to action does not always happen . 
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(Respondent: FLORALINE STEVENS) 

This year in anticipation of the Title I funding cutbacks, ' the number of Title I 
evaluators was reduced 'from 17 to four. However, most of these evaluators. were, 
placed in other units of the research and evaluation office. -It was recommended 
by a committee of school principals that the Title I evaluation budget be cut 
whiU at the same time the principalsVe re sti11 asking for evaluation services— a 
paradox! WheFTFoposition 13 became a\aw in California, evaluation services were 
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drastically red ue&d; however court-ordered , evaluation and federal compliance*, 
reports 'increased the need for staffing above the original numbers.* The initial 
.reaction of the superintendent is to cut and *then when the information needed is 
not available, some petitions or whole units within : our office are restored* 
There is another evaluation uiit that reports to the* Board of Education-, that* 
provides information on budget gats. 4 1 



■3 



IS 



* f Page 11 

(Questioner: CAROL H. WEISS) 

4. What advice would* -you give to an evaluator beginning work in an LEA? 
(Respondent: .STEVEN FRANKED 

Come in, with as broad an academic background as possible. Oir ideal candidate has 
a strong ba&kgrowd in measurement and statistics; a good background in 
accounting aid finance; and is adept in FOBTRAN and using statistical packages 
such as SPSS. *By far the "best minor area to have at this time is, business' 
administration, with at least two courses each in accounting, finance, management, 
and operations research. | 

Ir\ terms of previous work experience, we insist that half our 0 hires have 
relatively recervt public school teaching experience. The others we like to # get 
frbm consulting companies or from' particularly strong university based research 
groups such as Pitt's LRDC. 

Learn to write clearly and simply, and to meet deadlines. The easiest way to y get 
fired is to be a poor or a slow writer. Consider meeting deadlines your 'personal 
religion, and especially in your firrsfc tvra years on the job, do anything necesspry 
to- meet your deadlines. j . 

Finally, welcome to the most relevant wor^Ld of research. We think you'll 'have 
fun,' and you will be" able to see the results of your efforts relatively quickly. 
.That's more than most consulting company or Federally based ^researchers can say. 

(Respondent: » FREDA HOLLEY) 4 { . 

Trv^o understand the context; try to understand who and what you are doing 
evaSiation for and how to have impact. Bring the best technical skills' to the job 
you possibly can. .Remember that the II. S* public education system is in extreme^ 
danger and be ready to do what you can. to make .it. as effective as it can be/ 

(Respondent: < .CARL NOVAK) % 

The greatest challenge facing a new evaluator is ^establishing credibility with 
administrators in the district. In part the new evaluator 1 s credibility will^ 
initially depehd on the credibility of the evaluation unit or the credibility of 
past evaluators.' This natural transfer can be either an asset or a liability. A 
major portion will however, depend on tow successful the evaluator is 'in (1) 
focusing evaluation on % key aspects of programs, {and (2) establishing a mutually 
supportive working relationship with key administrators »'and other personnel. I 
can almost guarantee that in our district the credibility will not come from the 
use of sophisticated techniques nor scholarly reports. The key elements are-** the 
extent, td which the evaluation is perceived as useful (or potentially Useful). and 
the evaluator is seen as helpful. -The attitude with which the evaluator 
approaches his charge is very 'irjpoftant. If the' evaluator is seen as concerned 
and trying to make a contributojorr^Wie .battle is half won. If,- the evaluator is 
seen as aloof, pedantic^etc he^ or she makes the ..task much more difficult. 
Consequently my advic^to^ the evaluator would be to spend at least- ,as^ much time 
and effort tryj^gH^understand the dynamics of the program and relating the study 
and resulU^roUiose dynamics as he. or she does on developing the ^ evaluation 
desigrv^ev eloping instruments, analyzing data or writing reports. If you have 
questions about the program don f t be afraid to^ask. However, and this m is very- 
important, remember your role; do not allow program staff to call the "shots" in 
the evaluation.- 'Your role is to conduct £he evaluation, fulfill this role. 
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(.Respondent : ELLEN PECHMAN)* . 
Prepare to be inventive, thick-skinned , warmly tenatious and, as one of my 
evaluators quipped, - "Be^ sure that your health insurance includes a psychiatric 
' r plan! ff V ; - , 1 

In a 'somewhat more serious vein, 1 think the new evaluator needs to learn quickly 
that evaluation impact occurs constantly and indirectly. Every action of) the 
evaluator molds the outcomes of the information obtained "and the eventual use of 
the data. The smallest interpersonal mismatch can have the most dramatic effects 
on the availability of the very, information neede4 to conduct the evaluation. A 
program which wants to sabotage an evaluation can easily and creatively do so. 

{Respondent: FLORACJNE STEVENS) . 

Because this LEA is a 'large urban sckfool district, there is a large 'pool of 
evaluation applicants from which to choose. The % tendency * of some .is to be 
satisfied and not^to increase" their professional skills. My advice for those 
persons who want to be indispensible is continue to accumulate skills particularly 
* in the area of computer programming and data systems as well as advanced research 
design. Another area to' continue to study is ethnographic or qualitative 
evaluation techniques. If there is a need to cut staff, it will be those persons 
with multiple skills vrtio wifl be continued in their positions. y 
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^Questioner: BRUCE THOMPSON) I . 

1 . At a recent AERA annual meeting Scriven suggested that evaluation .would become 
a more potent influence on policy as more evaluators moved from' staff 
• positions into ' l^ne positions. Do you believe that" evaluators .are 
increasingly moving into line positions within Local Education Agencies? How 
- do you expect the careers o£ evaluators to fare in LEA settings in^he future? 

♦ • 

(Respondent: STEVEN FRANKED / ^ 

I don't think that there is any question that evaluation Will be increasingly 
valued, and become a more potent influence, as more evaluatorsget promoted from 
managing the evaluation init to. managing a wider range of funcj^fhs. , If nothing 
else, the fohner evaluators will be comfortable with* evaluation techniques and 
wpLll be able to tell the .difference between high quality and mediocre work. This 
in turn should lead to better management of research units and the productioh of 
better quality products which the former evaluation directors will know fully how- 
to exploit. 

,As for how often these promotions are occurring, Pm not really sure. ( In host 
'school systems, assuming a much wider range of responsibilities—including line 
ones— is usually the only way for the director of the evaluation department to 
become an assistant or associate superintendent. Therefore, given the high level' 
of competency and the, awareneap of system wide problems which many evaluation 
directors possess, one would expect that these promotions are occurring. Of 
course, Stu Rankin here on the platform is one example; Mike Say and .Bill Webster 
are others and Jim Jacobs in * Cincinatti has made it all • the way *to 
Superintendent. How often this is happening in smaller districts I. can't say. 

\ • . .. 

Evaluators 1 careers should do very well in LEA T s » As mentioned in a previous- 
response, tfii^ t may well be the fastest growth area in educational research since 
it's not dependent upon federal .funding and the worst effects of the. decline in 
birtfcs has already taken its toll. ..Also, being in charge of a school system's 
evaluation. .unit--, especially in a large district— provides a perspective on a 
school system 1 s operations that is. usually. only available to persons with the rank 
of assistant superintendent and above. This in turn should lead both tq . 
romotional oRportunites, and to "very strong feelings of Job satifaction and 
self-worth. All I can say is that in our shop, despite the. fact we pay at least 
20% ' less than the consulting firms and < government agencies which permeate 
Washington, we have not hed art opening for a full time research position in the 
past two years. In fact even before the federal funds dried up— we were getting 
several calls a week from experienced ' and respected researchers interested in 
coming to work for 'us. 

(Respondent: FREDA HOLLEY) ] » ' 

Although I am aware that a few key members of the R & E community have moved into 
superintend ancles, I do not perceive that this constitutes much of a trend. For 
one thing', most evaluators or researchers do not meet., certification requirements 
that permit them to move into line administrative positions, thus, the nunbers 
are not there for significant impact. Also, I'm not sureN^hat evaluators in * line 
positions will behava all that differently from^ the Sharp administrators we 
already have out there. I, believe that the incorporation of X£y' evaluation skills 
and concepts in administrative training programs will be muchi^ore the key po the 
long range change of school administration. * , \. 

18 ■ t • • " 1 . 
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(Respondent*: CARL NOVAK) 

This is a difficult question. I think it is too early to tell what happen^ to 
"old" evaluators (in terms of% career progression) because the movement is* tfto 
young. I do know of some evaluators who have moved into what I consider to * be 
"line" positions; however, I'm- not sure of the circumstances associated witn 
these moves. There could be any number of reasons for the moves including * being 
fed up with evaluation, increased jdb security, career advancement or mandatory 
transfer as a result of cutbacks. It is also important to remember that many 
individuals who initially held evaluation positions in LEAs were not trained as 
evaluators. The fact that they became evaluators is prima* facie evidence of a 
deviant career pattern. 

V -< . 

I also feel that individuals who have been trained' in evaluation and have 
functioned successfully as an. , aval ua tor in a supportive environment will be 
reluctant to voluntarily transfer to line positions. I* can understand why this 
might happen (money, jqj? gecurity, prestige, etc.) however I don't think it will 
be normal. The orientation and working conditions are too different. Lack ofD 
opportunity for career 'advancement for evaluators is and will continue to be a 
problem at least in smaller districts. 

(Respondent: ELLEN PECHMAN) 
\ It is very difficult to generalize on this issue from my experience. In New 
\ Orleans, it would be unlikely for an evaluator to be put into a line position 
since* our evaluators have limited school administrative experience and line staff 
are generally selected from the school site administrators 1 pool. Also, I find 
that people with the instinct for work with data .analysis and the technical 
expertise required of a good evaluator often aren-'t comfortable making the 
pragmatic. aid political compromises required of line staff. Maiay of us who are 
evaluators are good at presenting Irtie data for substantive arguments, but yre don't 
much care for the rapid fire*decision-making 7 -often without .factual data— often 
required of theMine staff. 



Regarding the next career stjep for our- evaluators, I think those who leave our 
evaluation unit' will likely move to similar^kinds of research jobs in other 
settings, perhaps in other governmental agencies, planning settings,/ or in 
research units in business or industry. However, we are really too smaHr^r group 
and we toavj^ existed for .too short a period for me to 'geqeralize with confidence 
regarding career ladders. * * 

(Respondent: FLORALINEJ STEVENS) / " . * ' " 

Only the top administrators of re'search and evaluation offices are moving into 
line positions; As* youjiote the titles of persons responsible for research and 
evaluation offices, you sometimes 'see listed "ponsultant"; § however, more are 
directors, executive directors, assistant superintendents and even associate 
superintendents. The number o£ evaluation staff positions appear. to be decreasing 
as' educational* resources dwindle and the mandates for> ongoing full-blown 
evaluation reports are no longer necessary. There will' always be a need for 
evaluation in LEA* T s but many of these needs may be fulfilled by ad hoc external 
evaluators instead of ongoing" full-time staf£. % 
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(Questioner:- BRUCE -THOMPSON) 1 
2. * In. a 1976 book ( Policy Analysts in the Bureaucracy ) , Arnold Meltsner suggested 
that' ev'aluartors could be categorized into four types, as follows: 



Type 
Technician * 


Skill Levels ' 
fcHigh technical, 
low political 


TABLE 1 
Typology of Evaluators 

CharactensticsJdentified By Meltsner- 

Motivated by opportunity to do research; judges success by 
peer approval; mam resource i$ command of detail; oriented , 
toward the long-term; sees analysis as an end in itself 




Politician • 

. \ 


^High political, 
low technical 


Motivated by desire for personal influence; judges succes^by^ 
approval of superiors who are perceived as being influential; 
main resources are communication and coordination skills; 
oriented toward short-term; sees analysis as a means for 
personal influence 


* 


Entrepreneur 

" A 7 

Pretender 1 


High technical, 
high political 


Motivated by opportunity to pursue policy preferences; judges 
success by policy impacts; main resources are knotyedge 
and communication sktfs; balanced time perspective, sees 
analysis as a means for policy influence and helping people 




Low technical, 
low political 


(Characteristics not iderjtified by Meltsner) 

H 9 





From EE PA , 1980, 2(2), p. 60. 

In your own experience, does this typology make sense? What is wtoml with it, 
if anything? What's right? - H • ' ^* 

(Respondent: STEVEN FRANKED , ' ' v / 

Without seeing the typology in context, it's difficult to comment upon it 

definitively. However, working just from the display provided to dS, f it raises ■ 
some questions. t 

""Let's look at management of evaluation units first. To successfully manage an 
evaluation njnit ip a large school system, one must have* both of the sets of 
characteristics which the chart lists for politicians and entrepreneurs. In fact v 
I would say that it is this combination of characteristics that, best 'describes the 
most successful managers. This immediately pleads to two problems with the 

-typology: * y ' ' 

1) The display implies that the politician ^ and' the entrepreneurial 
characteristics are mutually exclusive; or at least that one is dominant over 
the other. I would argue that these often go together; and that they are so 
Intertwined that^ they cannot 'be separated. 

2) The typology differentiates- politicians from entrepreneurs in relation to 
their skills by saying that politicians are often "low technical" whereas 
entrepreneurs are "high technical." I think this is clearly wrong. - Having 

. worked for four .educational consulting firms, and having been an officer of 
one of the most successful, I would say that almost invariably the most 
successful entrepreneurs are either "low technical" in almost every sense of 
the word; or else, they have made a conscious effort, not to employ their 

ERiC ' \ 20 ' \ : 
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technical background, no matter how well trained they^are. Why? -Because the^ 
♦truly' successful entrepreneur^ always thinking ahead to how the* study 
results can be exploited, ~" 



or where he/she is taking the organization next. 



Such people (jlon't have -time to worry about technical matters. That's what 
. \they pay their staffs to do v 

• ^ 

Conversely the reverse is often true of individuals classified by the typology 
as politicians. Often these are people who are very .technically oriented and 
superbly trained, and they achieve their personal influence by always being' 
able to provide a sound technical answer that .is relevant to the Issue at 
hand . - • 

Thus I* think* that general izatians regarding % the possession of technical 
ability are, dangerous. Th'e composite politician/entrepreneur, wftictyT^aim * 
makes the best manager, does not need much technical ability. w Further, - even 
if . he/she possesses it,' ^exercise of * these skills will often be 
counterproductive for the organization wheji viewed -from the opportunity cost 
perspective. , ' / 

Looking telow the management level , I think _that ; breaking into education 
evaluation as a' profession is most easUy done* by t;hose v*iich 'the typology 
classifies as technician. That's* vrtiat most research -direfctors want when they 
make hiring; decisions. Perhaps this is ' because the saccessful 
politicians/ entrepreneurs mu?t depend upon the technigians to do most * of the 
actual work. Hbwever, to go beyond the journeyman level, into management, I would 
sjrgXie that this perspective is best % left behind . • . - \ 

*As for pretenders, don't knock them. ,They are needed to collect and edit data, to 
compile statistics and make' routine computer runs, and to d6 the day-to-day tasks 

. that the Qther three types are resistant to doing. The trick is, you have^ to «be 

* sure that their salary doesn't grow faster than their worth to : the organization. 

' A dot of companies have priced themselves out of the market by paying 
attention to this fact. 4 „ * • 

• . / x v 

(RespQndent: FREDA, HOLLEY) m - ^ * \ 

Typologies such as this mgy be interesting and have some academic vstye; however, 
they are not very helpful in a practical sense. # '\ 

' ' V. . 

For one thing, such a classification schema has ( the implication of permanence or 

rigidity that is realty not present inhuman behavior. That is, an insect can be 
classified permanently as "djnaus plexipus" or a plant as a,"carfssa grandiflora" 
and the characteristics identified.. But $ person is rarely so easy to pin down as 
"high political, low technical." Let me use myseLf as an example. I am confident 
that I would have been classified as "low political ,$iigh technical" when I 
entered this arena using this schema, but over time I've learned that -politics are 
so crucial that I've moved more"ijv that direction and acquired political* skills. . 

* * * 
More helpful, iHhink, is a schema whereby skills that are important in evaluation 
are identified along wit*) the consequences for performance that, likely .follow from, 
the various pterforraance levels on that skill. It is Jaelpful to understand the 
role of political .skill- in the evaluator's work. Such a schema can be very useful 
,in a practical way then for training, for assessing how' an evaluation activity is 

, Xikely to proceed gi^en certain staff skill levers, and for "dec idirfg what . 
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supplementary resources may be necessary if policy impact is to. rtfsult. 

* *» .• " • ' 

(Respondent > CARL NOVAK) ' • „ > 

The typology is an interesting one and' provides a potentially insightful 
perspective for .classifying evaluators.' It is\ easy • to thmk of example^ of 
evaluators who ha^e a jfclitioal orientation. Urvforttjnately, it 1 $ all too easy to 
think of "evaluators" t who are neither technically competent nor politically 
astute. The entrepreneur classification represeQjbs the ideal or standard. 4 If an 
evaluator is to be successful he^or she *mu§t be technically cotppet$nt*and 
^politically insightful. . - . ' . \ * . " + 

* " , s y ° 

The strength of theHypology is that it points? out two. very 'important ^dimensions 
of evaluation (J) producing technically sound information, and (2) communicating 
that information and influencing policy. The typology poignantly makes- the *point 
that either by itself is insufficient: /The weakness of the ^typology is that it 
treats each dimension as a dichotomy^ Both are continua. -A /second problem is 
that the descriptors "high" and •Low" are relative.,. What is, by evaluators 1 
standards, low technical/ is probably from a principal 's^fTame of reference highly 
technical. Similarly high political may meap- $>ne. thing to an 'evaluator 'and 
another thing to an. assistant superintendent' or r ' personnel . dealing simultaneously 
with the board of education and the teacher's association, v 

(Respondent: .ELLEN PECHMAN) ' 1 

The Meltsner typology is an intriguing one that most certainly describes various 
".types" one finds Ifi* -school bureaucracies* but J don't thintf there are enough 
parallels between school systems and, the policy 9 planning agencies to make 
Meltsner 1 s framework fit school systems well, /^valuators ten<^ # to be teachers or 
teacher^types and these "types" aren't described by Meltsner's framework. Once 
evaluators becojne^ involved in "the . bureaucratic environment they adopt various 
organizational behavior patterns. Howeyer,* I tjiink their technical- and\ political, 
styles fall more on a continuum, with dimensions extending - from pretender to some 
well-developed combination of' political and „ technic al , than 'within a* typology. 4 

(Respondent:' FLORALINE STEVENS) \ \ . 

In my experience as- an evaluator in an LEA this doe5" not make sense. All 
descriptions appear* to'be' negatively oriented. I really resent the titles of 
technician, politician, entrepreneur; and pretender to describe a body of 
professionals/ just as doctors and lawyers resent the terms of "habks" and 
"quacks." My personal belief is that most evaluation professionals enter the field 
because of a strong interest in discovery afld not fojf monetary 'gain.' £ have met 
.few evaluators who have become rich or the top administrator of an LEA. In many 
instances the position is a' hot seat or necessitates walking a Eight rope. 
Technicians, politicians, entrepreneurs and pretenders. a&e <"too .smart" to be^in 
the situation-Kdescribed . S . ' • * 
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(Questioner: BRUCE THOMPSON) * 
3. * For several years the ' literature on evaluation use suggested that LEA 
administrators," "under use" evaluation information. For example, Krlbourne and 
DeGracie (1979 AERA paper, p. 12V suggest that: 

All LEAs, with possibly a few exceptions, can point to 
^ , their, ^o^umes of research and evaluation verbiage setting 

on the shelves of district administrators being used for 
little el.se than a door stop, swatting flies, or any of 
/•the other various and sundry purposes for which research * 
\ is 'used in the public schools. C^-^ 
x However, recently a more optimistic tone regarding use -levels has become 
* popular (e.g., ED .170 ,345). For example, it has been suggested that there are 
several types of use, including use vrtiich -affects administrators 1 "ways" of 
thinking about .programs" rather than directly affecting specific,' policy 
decisions (ED 199.271, pp. 11-17)-: °°es this distinction between types of use 
v make sense to. you? What are the most- typical forms of uSe' in your job 
setting? 

(Respondent: STEVEN FRANKED ' - \ J 

If evaluation results are under used, it is usually the fault of the evaluators 

and their managers. ' 

First, eveijy study- should be presold. It should not; be done unles^ there is a 
demand for its results, and a definite chance that v it may serve a useful purpose. 

Second, every study' should be designed so as to maximize the chance that it will 
provide administrators with information % which they can 'act upon. If it jbecopies 
apparent that this goai cannot be, met, the study should be quickly ^cut back or* 
cancelled. Third, the '.results must be presented in a manner which is both 
readable and useful; and it should be* oriented towards policy makers and 

^citizen's/ In our f cas£, this increasingly means that our- reports follow the format 
of going fVom no -more than a 1-2 page introduction which cites why the study was 
conducted^ aid who asked for it, to a polidy oriented executive summary, and then 

^to a detailed discussion of results. We then put everything else— including the 
stijdy design, sampling plan, instruments, and analyse^-- into technical appendices. 
In a large study we bind these appendices into a separate volume. The value of 

* this approach is proven by the facft that for a typical -study, we often "sell out" 
our initial run of 300 copies of the main report whereas the ; -50 copies of the 
technical appendices often go begging. Vfe also produce the executive summary 

" under separate cover , and demand for this has o'ften led to more than 500 -being 
printed . ^ . * . 

I " ' ' . * 

I don't like the idea of trying to use - evaluation reports to "change the way 
administrators think about programs" as opposed to trying to impact upon specific 
policy decisions. I would view, the former as the coward's way t>ut, and would much 
rather try to produce reports which are so relevant that -the § admin istr a tor-can 

* only ignore them at his/her own peril. 

(Respondent: FREDA H0LLEY) . 

I concur with the basic premise here, that evaluation use is not direct and easy to 
oBserve. Neither, however;* can it be capturedVwith any other single notion. 
Rather, use is determined by a myriad number of factors which create all sorts of 
jjse configurations, any one' of which <,is as likely to resemble the n|xt as one 
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snowflake is to resemble another. In pur district^ some uses of evaluation 
findings are immediate and direct, 'some emerge after the initial, findings as 
"truths" belonging to the lore tof the district. I feel very upbeat about "what can 
happen based or*, our experience. ' I think >it would be terribly important to 
education if. superintendents and school boards could be convinced of what 
evaLuatipn can do. \^ * - 1 

(Respondent: "CARL NOVAK) • 

It is very difficult to identify when specific polity decisions are actually made 
,and even more difficult to * sort out alL the bits of information that contribute to 
that decision. T<j.me, the alternative ways of explaining how evaluation results 
fere used make sense. Administrators <v*io are held responsible for the decisions 
they make and for the outcomes. associated 'with those decisions are not going .to 
default that responsibility by deferring to the results of a single evaluation, 
particularly if the findings suggest" an action that t runs counter to past 
experience^ intuition. However , the outcomes of the evaluation should clause the 
administrator to -think differently about the decision than if he or she -had no^ 
knowledge about- .the outcomes. Just because* the results .are negative toward the 
most- appealing d^cjLsion alternative doesn't mean that the alternative won't \e 
selected* ' It should change .the decision making process and the likelihood that 
the impact of the decision, will be more 'closely monitored. ' The fact that 
evaluation was conducted, irrespective Of the results, probably changes the way % 
decision makers think about a program. Use, of course^ is ^ also af fected v by 
number of other factory, *.some of which ace j.dentified iff question 2. 

The typical format for * presenting results in our* district includes oral 
presentation of the re$ults v v &o the superintendents cabinet (or some decision- 
making, body) , . accompanied by an executive summary or dlher handouts ('for 
references), and followed by an unstructured discussion of the implication of the 
results for the program, The fofm^it is -designed to promote^ use. A fantasy .of. 
mine regarding use of results is that policy decisions ^administrative option 
are either' justified, in terms of available evaluation results *or -if the decision 
at action is inconsistent with the results a compelling rationale is developed as 
tS why tha^ particular decision or action was felt to be appropriate. This is 
still nj3inly a fantasy. . , 

(Respondent: ELLEN PECHMAN) * •„ ••• , 

In a recent presentation at NIE, . Mary Kennedy, described^ school administrative 
behayior as based upon administrators 1 -' "working knowledge." Working knowledge, - 
according to Kennedy, ••consists of things we 'believe to be true; it is intuitive, 
sensitive to change «and relevant to the working environment. "Working knowledge" 
refers to the technical information a person-uses, molded and recast into the 
language and circumstance of that individual. Kennedy has fSund that school 
people uniquely apply technical information in a form that is modified accord^bg 
to their specia l cirg jumstanbes, past experiences and interpretations of the facts 
they observe. Thus, to the*6xtent~tiiat the data reported in an ; evaluation are 
consistent, with the working knowledge of the user, it may 'influence any number of 
aspepts of an administrator's ibehavior. Certainly^ the rational approach to 
evaluation use—evaluation takes place, is reported, data are considered and a- 
decision is made-- is the least l^Lkely^eK ample "of how evaluations are actually used 
in % NeV 'Orleans. The extreme opposite example of this traditional view of 
evaluation use occurs often— & program will finally get under Way or will create 
its own ^fbuation data to avoid being evaluated. Is this evaluation use? ^ 
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It seems there are two concerns: (1) the theoretical problem of defining 
evaluation use for" the purposes of research and k greater understanding of 
evaluation behavior and (2) the practical question of what kind of datfa , can be 
used to improve educational decision-making (if .any); ^^nd how should it be 
* / presented? / * -—^ 

' I am hard pressed to conceive of a future where we will be able to develop a neat 
framework or -typology of evaluation use that describes the phenomenon, which occurs 
regarding evaluations j.n schools in a practical^and useful way. Oi one hand, I 
certainly think it is necessary for us to continue to *ex-plore~ the niinerous 
"elements 'and actors which interact in decision-making in schools' so that we can 
improve the quality and effectiveness of the data "that are used to make important 

/ ed Rational plans. I 'am currently thinking about evaluation use in terms^ of a 
cubic, matrix in which the user , ther evaluation context , and the data^ form three 
'sides of the cubic structure. Each of the three structural elements has numerous 
subelements which cross and create numerous new "types" of use. The framework I 
am describing, might look iike the one below: " * 

/USER 7! 
/ / i 
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I am hard-pressed to describe the typical form of evaluation in our district, 
'since each evaluation const itiit£5__a_ slightly different example of^type of use. 
Use is a highly individual experience. One of our key administrators recently 
, told us that he believed evaluation reports should present 'the data, report the - 
statistical significance, and state, the "educational significance" of the report. 
But he emphasized that he didn't believe the evaluator should offer any, 
' recommendations or conclusions about programs. How or -if to use evaluation 
information, according to this administrator, is entirely the decision-maker's 
prerogative. If this is the common view, the issue of improving use may be out* of 
the* evaluator 's hands unless we attempt, to trick people into -use of 6ur findings 
through skillful public relations gimmicks. * 

(Respondent: FIDRALINE STEVENS) 

Evaluations mandated by remote (state /federal) agencies generally -have little 
effect".- I refer specifically to compensatory education and Title I evaluations 
and their reports. However, compliance evaluation reports or court-mandated 
evaluation reports which expect the v data to point out areas of need , .areas 
receiving' some kind of mediation, and are* successfully dealt with are used 
x frequently and meaningfully. For instance, my school district is under a ^ 
^ court r or8ered desegregation plan. The .court has ordered the evaluation of the 4 
district's programs which -were planned to • ameliorate the harms of racial 
isolation*. Program personnel have used^ the report information to satisfy the *• 
court as well as to make changes to Improve the programs. This is also true of 
"the LAU evaluation reports to. meet the compliance issues coming • from the Office • 
r. for Qivil Rights. - 
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It has been ray experience that process/progress/ formative 'evaluation reports have 
been judged" -to, be useful to program developers and implemfenters . Sgmmative 
evaluations—number crunchers — are viewed as useless or too ^ late for impact." 
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(Questioner: BRUCE THOMPSON) s 
4. Several persons "have suggested that evaluators would be more effective if they 
dropped their cloak of unachievable objectivity and actively tried to optimize 
use of evaluation. It has even been suggested that evaluation really ought to 
be considered persuasion. How far do you think evaluators should go in trying 

* to optimize evaluation use? What danger 3 are inherent in the various 

* decisions whiph can be taken in this regard? 



(Respondent: >STEVEN FRANKEL) * - 

My answer to this rquestion is largely contained in my response 
question. Unless we try to optimize evaluation use, we 
Furthej>f if we "can H optimize use,' we don't deserv'e to survive. 



to the 
will not 



previous 
suryive. 



As* to 'tow far one should go, I would say very far. 
get, ydur 'Board to implement a policy stating that: 
activities must be made public and distributed to 
citizens and newspapers^/* the community; and 2) 
regarding individuaL^cpnfidentiality, all computer 



The first thing to do_ is to 
1) All results from evaluation 
'both the Board and to key 
that after safeguards are taken 
files and printouts will be 



made avail abLe-^to^Siyoi^' T (». a**c8st reimbyrsstlble basis. How does this relate to 
creating "a "demand for evaluation results? ^Simftle. As noted earlier, in a well 
run operation no [evaluation should be 'conducted unless a reasonable level of 
controversy exists /regarding the program. What *his, policy does is insure that 
individuals on bdth sides of therissue will haMhave full access to the study's 
results and recommendations. This in turn creat 

day. the study i k s * initiate 
as people learn that no matter 
it will produce'* some results. use 



results from the/ 
additional studj 
eventually favol 



1A 

The major danger that must be m avoided is con 
responsibilities. For evaluators to remain objectiv 
for implementing their recommendations, ^e minute tW 
doing % 'so, they have co-opted themselves for future st^ud: 
'As long as they assess, recommend, and criticize they are 
begin showing others to'w something ^should be done, they ar 




a> .built-in demand for the 
It also creates a demand for 
£h side of an issue a study 
tb both sides. 




Another danger is in being reluctant to change your mind, or 
wrong. A given percent of the time, taking the advice of evcfiL 
disappointing* results. At this point, all you can do j.s point o 
the superintendent and "Board remember, your overall win vs. loss 
your mistake. 



staff with line 
cannot be responsible 
caught in the trap of 
in that program area. 
; the minute they 
deep trouble. * 



that you - were 
s -will lead to 
hope thalfcj- 
and admit' < 



r^gs of 
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(Respondent: FREDA HOLLEY) 

Few experienced evaluators I know wear cloaks of any type; tat 
hard-won t experience maybe! 1° belong to the "persuasion"* ranks , b 
to^iqc.lude evaluation findings, research findings, and in fact 
information in educational ^decisions. I do not believe it our 
demand decisions which conform to an> particular source of information, 
it is folly not to recognize that today's world of school administr 
instruction is in a* state of information overload. Therefore,^ it 
competition and persHasion simply to get the ear and eye of the audience , 
them attend* to and .understand the findings. ~T~ 'personally *a» pleased 
educators who can weigh evaluation results with political factors, with intui 
and with other sources* of information to arrive at formal judgments. 
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Decisions made with a lack of- knowledge are the real ignomity^of ^ucatio n^ -Where 
they .occiir aid I had J<nowl edge that I -fai^etf rfcr - ^^ 
-and the burden of responsibility. 4 ' - 

'(Responent: CARL NOVAK) * 

After carefully considering the question I now feel that ifc . deserves an 

unqualifiable and inexplanable "possibly." % . ' 

While complete objectivity might be unobtainable, openness is. not. The, most 
satisfactory evaluation (most Useful, most credible, etc.) is one in which a large 
number of people have had input. Initial involvement -of program'' and/ or 
administrative staff in the study eliminates many'of the problems in .disseminating 
the results of the study. More importantly, familiarity with the* evaluation 
increases understanding of the effort and the relationship of the evaluation and 
the program. *[t 'also results in the early identification of controversial aspects 
of the evaluation (in time to resolve conflicts 'before the results become 
available and ideally even before the data \are collected). Lastly, earlier 
involvement .increases the likelihood that the results can and wil^L be used., 

Early involvement of interested parties in the evaluation puts the evaluator under' 
pressure to justify choices in design, instruments, etc., and to 'give due 
consideration" to competitive points of view. It does not change the evaluator's 
role in the reporting process. Valuators can and often do have very different 
perspectives of the program they are evaluating than do teachers- and 
administrators. . The evaluator is responsible for ^ representing Ijis viewpoint as 
fairly and faithfully as possible. Persuasion is inevitable in 4 the reporting 
process. The evaluatgr ethically must represent bis viewpoint but she of he must 
do so 'as openly and as fairly as possible. He must ^always be concerned^ aboufc his 
motives. There is danger that the evaluator promotes his point of view for the 
wrong reason or with an inappropriate constituency^ i.e.,* worries more about 
personal^ power and prestige (winning) than in helping the decision makers achieve 
a more balanced understanding of their 'program. * 

In summary, evaluators ^should pfomote the use of evaluation results but they must 
do so as openly and ethically as-ipossible and for the right reasons. They must.be 
perceived as^^helpful- rather .wan as advocates or adversaries. Continuing 
relationships with constituencies make openness, candor and mutual respect 
indispensable. 

(Respondent: ELLEN PECHMAN) t * 

Over the years I have lost the illusion that ' there is such a fching as an 
"objective fact" in the school world. Instead I have begun to thinflfin terms of 
numerous possible objectivities ♦ .Too frequently the same set of data *lead to 
entirely different conclusions, recommendations, or ^terminations regarding the 
actions implied. Thus, I believe that evaluation is always persuasion. In the 
best circumstances, the effort" to persuade a potential user that the data 
accurately reflect the key occurences in the evaluated setting is worthwhile 
because the data are, indeed, accurate. If optimizing 'use means improving the 
working knowledge of the user, the evaluator shoul d go a long way. But when data 
or events are distorted out of .proportion or made so . simple_that the event 



described no longer resembles what occurred, we have gone too far. Dangers occur 
only when extremes are dominant .j Neither evaluators . nor their data are 
omnipotent; claiming objective purity or drawing conclusions without ' sufficient 
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data-tfesed • support are both significant professional mistakes ^valuators can make. 
If evaLuators .haue the personal Skills and the intellectual integrity to determine 
wtiere,' and when to take a stantf—usirtg their data honestly and ( well—use of 
evaluation 'will occur as well as*the use ponfext willf allow. 

% (Respondent: FLORALINE STEVENS) h ' ~* 

In my school district there is already much interaction between evaluators and 
program staff because most of the evaluation is formative/process-oriented. There 
is no need to'wear an objective clodK but to -.be professional and open about 
various biases or values held. Evaluation should, not be that of persuasion but it 
should * point up discrepancies between falan and reality/ It is still up to the 
program persons to decide what to^do about any discrepancies. Evaluktora should 
Step back after reporting the in formation » in the report. I dp not believe they 
Should "sell" their recommendations.' I honestly do npt believe that a person pan 
even tidily- do a summative evaluation of»vfaat he/she has, planned .if he/she^ has 
-"persuaded" program people to follow a certain recommendation in a particular way. 
This is the danger vfceh an evaluator persuades or sells a certain step. There . \s 
less danger to recommend based upon a finding ,and leave to the program person the 
plan to develop : and implement as a response to the" recPmmendatibn, - if the 
recojnmendation is believed,, to be gopjd or valid. w 
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(Questioner: BRUCE THOMPSON)^ x 

5. National surveys of LEA evaluation personnel indicate that many LEA evaluators 
have not taught. ' In your o^ion, would lack of teaching experience 
drastically lessen evaluators' edibility in the e y es of most LE ^ 
administrators? 

(Respondent; STEVEN FRANKED 
I would say that it certainly doesn't help it, N t^ut a lack of teaching experience 
is certainly not fatal. In our operation we hawe publicly committed, to alternate 
in our hiring between those with teaching backgrounds and tbose with consulting 
.experience. This has' worked very well^ both *^from an operational .and. pyblic 
♦ • relations standpoint. Also, we have tried to makg our^eaching hires from within 
the 'school system. This has permitted us to hire a Black math teacher that we 
have trained as a researcher; was the former financial account manager for air 
special education programs who we now use to-aanage evaluation contracts in tt^it 
area; and a former Fullbright scholar vho had to give up teaching because her 
health no longer permitted her to stand in front of a class. , Interestingly of our 
three managers, only one has public school teaching experienoe. 

.<«- However, I would be just as worried if an .evaluation unit was staffed completely 
by those wyth teaching backgrounds.' I think that having persons with business 
' and/ or consulting backgrounds is just as impbrtant as having people with teaching < 
backgrounds. In fact, I would say that more evaluation shops in school systems 
suffer from a lack of nonteaching, backgrounds^ than from not having sufficient 
-personnel with teaching experience. 

(Respondent: FREDA HOLLEY) 

If evaluators demonstrate a lack of experience or reveal a lack of experience 
whether of teaching or of administration, it .will naturally hurt their 
credibility. So will any lack of evaluation expertise. So will - dressing 
inappropriately or maintaining an aloof "-attitude. 

* 1 z * * * 

• I have had evaluation staff without teaching experience, however, who were highly 
credible and highly regarded. . Again, there are no invariable rules about such 
things. Personally^ I had brief teaching experience at 'both the elementary and 
k secondary level. I had experience also"- teaching more than qne subject. I believe 
that this experience was valuable and that it has helped -,me understand many 
evaluation situations and problems with -lejss effort. Oi the other hand, -L 4 am 
aware that conditions today are not like those in which my experience occurred nor 
does experience in one type situation necessarily transfer to another. Again, 
• what I'm saying is that perceptiveness , empathy, and intelligence compensate for 
lack 'of experience;- but experience- will be. no substitute for the fortner 
* c k characteristics. * m 



. (Respondent: CARL NOVAK) * . , . ... * . 

No, not lack of .teaching experience peruse • Particularly rxjt in the long run. 
Credibility comes from (1) effective evaluator-client relationships, which in turn 
depends upon sensitivity, a sharing of purpose (commonality 'of purpose) and an 
understanding of program/ school 'dynamics, and (2) soundness of methodology. 
Neither requires teaching experience. ' 



The above 'statement should not be interpreted to mean the teaching experience is 
not valuable, I propose- that teaching experience. can be important. Individuals 
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, who have had successful classroom experience often have a better understanding of 
the dynamics of' the classroom and are more attuned to howtschools function. They 

.very often have a different set of values and different attitudes about school 
than individuals .who become* evaluators without the benefit of practical 
educational experience. They frequently have different career goals. N 

• Consequently, they are more able to empathize with teachers and establish better 
relationships with administrators. Simply put, they have more in common and % can 
establish better relationships which lead to improved credibility ( if methodology 

1 and other factors are held oonstant) . Unfortunately it is also possible that 
individuals can have teaching without gaining appreciably from it. 

In 'summary, while it is often valuable for 4 an evaluator to -haVe had successful , 
teaching experience, it is more important* that , they have good interpersonal 
skills, are perceptive' about organizational * dynamics and have good 
technical/methodological backgrounds. Teaching experience won't compensate for 
, . lack,of competence or inappropriate attitudes* 

(Respondent: ELLEN PEdHMAN) ' 
One of the major qualities an evaluator needs is an ability to comprehend -the^ 
context evaluated. . Without a complete understanding of 'the people, the. situation 
•and the cross -current of the events to be evaluated, it is difficult for an 
. evaluator to ask the kinds of questions needed to obtain the data that accurately 
reflect the program or situation assessed. Thus, it helps for an evaluator 'to 
.know how it* feels tp. wear the teacher 1 sr* or the director's shoes, but such 
. sensitivity may be available through other means than prior experience in the same 
situation. It largely depends upon a person; s insight and general intellectual 
skills., When evaluators 1 credibilities are questioned, it is because they have 
failed upon initial meetings or encounters to represent themselves well. Prior 
teaching experience will onlyi be necessary to establish credibility if an 
evaluator is naive in developing the working relationships and fails to convince 
\the client that he or she can represent the program well . I think v evaluators need 
• Aplenty of' experience with schools to understand how to find adequate- ways to 
evaluate educational programs, but 'the precise nature of the experience can vary 
t . depend ing^upon the skills of an individual evaluator. c 

(RespondenC: FLORALINE STEVENS) 
-My response to this question is two-prortged . I believe that in a small school 
; district t where the organizational structure or levels are not very complicated or 

complex, an evsCLuator without teaching experience could provide the services 

needed and - thus have credibility with most LEA, administrators. However,. if this m 
"person had the assistance of someone knowledgeable about schools working with 

him/her, the credibility would increase. 

Now in a large urban school district with many levels of administration and vast 
-numbers of schools,— * the task* of evaluation becomes quite complex. An evaluator 
without school* experience in this milieu would have a difficult time establishing 
credibility in the eyes of the LEA administrators in the beginning . However,, as* 
'the evaluator undertakes more assignments aqd thus has more contacts with LEA 
administrators, the experential base will improve. When the evaluator begins to 
Bpea k th e la nguag e of L EA-administrators, his/her credibility will be in place. 
^This is a ' process that takes time. Certainly assistance or - partnership with an 
^internal LEA person aids in the process.' The .question is how mUch time -does — the — 
"evaluator have to accomplish credibility with LEA administrators. 
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The ESU 1 8 Evaluation Team was organized in 1 973 to provrde evaluation support to 
'the^ederally funded programs being operated within the district that had- mandated 
evaluation requirements. Prior to 1973 the director of each federal l/funded project had 
to independently make arrangements for evaluation. Alternatives available to the 
directors included sharing an evaluator among several projects, individually-contracting 
with an external evaluator (typically either a university professor or a consulting firm) or 
assigning evaluation responsibility on a part-time basis to a staff member with no formal 
training or experience in evaluation. The evaluation team made it possible to coordinate 
the evaluation of all the federally funded projects within the district. In addition to 
efficiently meeting the mandated evaluation requirements, the evaluation team concept 
4 also provided the district with evaluation support forjocally funded programs. 

Although the evaluation team was organized primarily to^valuate federally funded 
programs, the role of the team in the district has changed. During 1973-74, the first year , 
the evaluation team was in existence, the team worked almost exclusively on federal 
programs. Only about ten percent of professional evaluator time was used to support 
locally funded programs. The proportion of time spent on local programs increased 
gradually from 1 973-74 through 1976-77. During the 1976-77 school year the proportion 
of time spent on local projects was approximately equql to the proportion of time spent on 
federally funded projects. Since then the proportion of time on district activities has 
continued to increase. During the 1980-81 school year the^Jroportion of evaluation team 
time devoted to support of local projects was approximately 2.5 times greater than the 
proportion devoted to federally funded projects. A comparison of the proportion of . 
evaluation team. workload devoted to federally funded and locally funded projects over the 
first eight years the evaluation team was in existence is presented graphically in Figure I. 
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The trend away from the evaluation of federally funded programs toward support of 

locally funded programs is well documented in Figure^ I. The reasons for the shift^re 

more complex. In' 1973^ when the evaluation team was Organized, the district was 

operating five large scaJe programs that had mandated evaluation requirements. The five 

were: Title I, Follow Through, two Title III projects (Project Instruct and the Learning 

Community Schools Project) and Career Education. As-the district began the 1 981-82 

school year only one of those projects, Title I, was-still in operation. Several new federal 

projects have been funded. However, these project's are typically smaller in scope than 

the projects funded in the early 70 f s.' Therefore, less time is required to evaluate these 

* smaller projects. Another,/probabl/even more irppprtant reasorf for the shift in emphasis 
is a diversification of the types of services that f he evaluation team provides. The team 
was initially organized to evaluate Vograms, that is to determine whether or not a 
program was effective. While it still maintains Vhis capability the •feam is now also - 
involved in interpretation and development of educational tests, community and staff 
surveys, information processing and use, support of curriculum studies, and consulting. 
The diversification of services provides the evaluation team with increased flexibility that 
enables tepm members to focus more directly on the information needed by project 
directors and administrators. For example, during the 1981-82 school year the evaluation 
team wijl work on approximately 40 to 45 different projecTs. Examples of types of 
projects include (I) Evaluation of such programs as the Hoffman-Gould Reading and Math. 
Program at West Lincoln and Arnold Elementary Schools, the Title IVc preschool 

C handicapped program, the use of microcomputers in the district and evaluation of 
processes of special Interest such as the implementation of the district reading program 
and progress reporting to parents; (2) Assessment and t^st related projects such as the 
development and validation oOpa{th assessment tests for both the junior high and 
elementary schools and the rei^Qn^^the senior high health waiver test;, (3) Community 
and/or staff surveys such as cohirniajity^quity survey, administration of a school climate 
profile at Everett Junior High School, and the survey of community attitudes toward 
Lincoln High School; fa) Support of district curriculum studies such as the Physical 
Education-Science, Music and Industrial Arts studies; (5) Special studies such as the 
district math progress*study and; ; (6)Xenerdt support ot eerifqaladmM 
• making enrollment projections for ^982-83, conducting p comprehensive inservice needs 
assessment of special education and developing a systemjhat uses multiple criteria to 
select students for Title I. / 

So far during the 1981-82 school year the evaluation team has published sixteen 
^evaluation report^ and a similar number of mini-reports or memorandd f or the record. 
The evaluation team plans to corttinufejto provide evaluation and related support to both 
federally funded and district programs, and to focus it's efforts on the priorities ^ 
established for It by % the-Superintendeot f s Cabinet. — - — — - 



